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DETAILED ACTION 
Response to Arguments 

1 . In view of the Appeal Brief filed on December 22, 2003, PROSECUTION IS HEREBY 
REOPENED. A new ground of rejection is set forth below. 

To avoid abandonment of the application, appellant must exercise one of the following 
two options: 

(1) file a reply under 37 CFR 1 . 1 11 (if this Office action is non-final) or a reply under 37 
CFR 1.113 (if this Office action is final); or, 

(2) request reinstatement of the appeal. 

If reinstatement of the appeal is requested, such request must be accompanied by a 
supplemental appeal brief, but no new amendments, affidavits (37 CFR 1.130, 1.131 or 1.132) or 
other evidence are permitted. See 37 CFR 1.193(b)(2). 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-25 are rejected under 35 U.S.C. 103(a) as being unpatentable over Johnson (US 
Patent No. 5,748,974) in view of Cote et al (US Patent No. 6,125,347). 

Johnson discloses a multi-modal natural language interface for cross-application tasks 
which interprets user requests combining natural language input from the user with information 
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selected from a current application and sends the request in the proper form to an appropriate 
auxiliary application for processing. 

Regarding claim 1, at col. 3, line 63 continuing to col. 4, line 20, Johnson teaches the 
system sends the output of a speech recognizer and non-speech input received by a screen 
manager to a dispatcher. Based on the output of the natural language processor, the dispatcher 
invokes the application manager to determine which application should process the request, 
which reads on "presenting a command associated with an application to a dialog manager, the 
application associated with the command being unknown to the dialog manager." 

Johnson fails to specifically disclose, "the application associated with the command being 
one of a plurality of active applications." However, implementation of a computer system, 
which allows a user to interact with a plurality of simultaneously running applications using 
natural language, was well known in the art. i 

hi a similar field of endeavor. Cote discloses a system for controlling multiple user 
application programs by spoken input, which provides a speech recognition program working in 
conjunction with an operating system which provides multi-tasking capability and provides for 
implementation of a windowing system which facilitates the direction of user input to the various 
user application programs which are running (col. 3, lines 12-17; col. 4, lines 36-39). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to allow for the presented command to be associated with one 
of a plurality of active appUcations, as suggested by Cote, for the purpose of allowing the user 
the convenience of accessing the processing of all applications to have a request processed 
automatically, without the user having to manually access any of the applications. 
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Johnson fails to specifically disclose "the dialog manager determining current context of 
the command by reviewing a multi-modal history of events." However, determining current 
context of user input responses by reviewing a history of events was well known in the art. 

Cote teaches the system maintains a database for the application programs that are 
running which allows the system to maintain an effective context for the user programs (col. 1, 
lines 47-49) and the system creates text events which are used to record information about all 
types of input events or application events (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement determining current context of user input 
responses by reviewing a history of events as suggested by Cote, for the purpose of providing an 
improved system for controlling multiple user application programs by spoken input, as also 
suggested by Cote (col. 1, lines 32-34). 

Additionally, at col. 4, lines 29-52, Johnson teaches the application manager determines 
which application is applicable to the user input, sends a request to a code generator for the 
application, and generates the application program code interface code required to access the 
information, which reads on "determining at least one method responsive to the command based 
on the current context." 

Additionally, at col. 4, lines 48-52, Johnson discloses a response generator that generates 
a response appropriate to the nature of the request submitted by the user and the current 
application, which reads on "executing the at least one method responsive to the command 
associated with the application." 
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Additionally, at col. 4, line 17-53 and col. 5, line 23-29, Johnson teaches "changing 
dialog focus," because the system ascertains that a different application should process the user 
request, and utilizes the different application's resources to process the request, since the 
procedures to access a new or different application requires the system to access or focus on the 
new or different application. The system of Johnson performs this processing without requiring 
the user to perform the task of accessing the application, but the application is indeed accessed 
and opened to allow for the processing of the user's request. 

Regarding claim 2, Johnson and Cote teach everything as claimed in claim 1 . 
Additionally, at col. 3, lines 33-42, Johnson teaches the multi-modal interface is linked to the 
applications and allows for combining input from various modalities, which reads on "employing 
at least one multi-modal device for presenting the command." 

Regarding claim 3, the combination of Johnson and Cote teaches everything as claimed 
in claim 2. Additionally, at col. 3, lines 37-42, Johnson discloses the input means can be via 
speech, typed or handwritten and includes point-and-click, touch and keyboard inputs, which 
reads on "multi-modal device is a computer." 

Regarding claim 4, the combination of Johnson and Cote teaches everything as claimed 
in claim 1 . Johnson does not specifically teach providing a linked list of all events in the multi- 
modal history. However, providing a linked history of dialog between a user and a system was 
well known in the art. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or application events (col. 7, line 9 to col. 9, line 53), which reads on 
"hnked list of all events." 
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Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 

Regarding claim 5, the combination of Johnson and Cote teach everything as claimed in 
claim 4. Johnson does not specifically teach events in a multi-modal history include events 
linked by time, type, transaction, class or dialog focus. However, providing linked history events 
of a dialog between a user and a natural language system was well known. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or application events, and the events or records have chronological 
and/or hierarchical relationships (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 

Regarding claim 6, the combination of Johnson and Cote teach everything as claimed in 
claim 1. Additionally, at col. 5, line 55 continuing to col. 6, line 19, Johnson teaches referencing 
a concept/application table in which for a given inpixt the application manager finds all concepts 
in the representation and determines each application registered with every concept in the 
representation to create an application set. The semantic representation is sent to the applications 
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code generator to allow the application to act on the code and retrieve the requested data, which 
reads on "referencing all active applications using a component control to determine the at least 
one method which is appropriate." 

Regarding claim 7, the combination of Johnson and Cote teach everything as claimed in 
claim 1. Additionally, at col. 6, lines 20-40, Johnson teaches the semantic representation of the 
users input is sent to the applications code generator to allow the application to act on the code 
and retrieve the requested data, which reads on "the command is presented in a formal language 
such that a plurality of human utterances represent an action to be taken." 

. Regarding claim 8, the combination of Johnson and Cote teaches everything as claimed 
in claim 1. Additionally, at col. 3, lies 60-62, Johnson teaches maintaining item in focus 
information, which reads on "maintaining a current dialog focus." At col. 5, line 61 continuing 
to col. 6, line 9, Johnson teaches concept/application table which reads on "a hst of expected 
responses to provide a reference for determining the current context." 

Regarding claim 9, the combination of Johnson and Cote teach everything as claimed in 
claim 1. Additionally, at col. 4, lines 9-17 and col. 5, lines 9-11, Johnson teaches the natural 
language processor interacts with the user to clarify missing information or ambiguities, which 
reads on "querying a user for one of information needed to resolve the current context and 
information needed to take an appropriate action. 

Regarding claim 10, at col. 2, line 62 continuing to col. 3, line 16, Johnson teaches the 
hardware environment used in the multi-modal system, which includes running an operating 
system supporting multi-tasking and includes a central processing unit and Ram for storing 
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application program code and data, which reads on "program storage device readable by 
machine, tangibly embodying a program of instructions executable by the machine." 

Additionally, at col. 3, line 63 continuing to col. 4, line 20, Johnson teaches the system 
sends the output of a speech recognizer and non-speech input received by a screen manager to a 
dispatcher. Based on the output of the natural language processor, the dispatcher invokes the 
application manager to determine v^hich application should process the request, which reads on 
"presenting a command associated with an application to a dialog manager, the application 
associated with the command being unknown to the dialog manager." 

Johnson fails to specifically disclose, "the application associated with the command being 
one of a plurality of active applications." However, implementation of a computer system, 
which allows a user to interact with a plurality of simultaneously running applications using 
natural language, was well known in the art. 

In a similar field of endeavor. Cote discloses a system for controlling multiple user 
application programs by spoken input, which provides a speech recognition program working in 
conjunction with an operating system which provides multi-tasking capability and provides for 
implementation of a windowing system which facilitates the direction of user input to the various 
user application programs which are running (col. 3, lines 12-17; col. 4, lines 36-39). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to allow for the presented command to be associated with one 
of a plurality of active applications, as suggested by Cote, for the purpose of allowing the user 
the convenience of accessing the processing of all applications to have a request processed 
automatically, without the user having to manually access any of the applications. 
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Johnson fails to specifically disclose "the dialog manager determining current context of 
the command by reviewing a multi-modal history of events." However, determining current 
context of user input responses by reviewing a history of events was well known in the art. 

Cote teaches the system maintains a database for the application programs that are 
running which allows the system to maintain an effective context for the user programs (col. 1, 
lines 47-49) and the system creates text events which are used to record information about all 
types of input events or application events (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement determining current context of user input 
responses by reviewing a history of events as suggested by Cote, for the purpose of providing an 
improved system for. controlling multiple user application programs by spoken input, as also 
suggested by Cote (col. 1, lines 32-34). 

Additionally, at col. 4, lines 29-52, Johnson teaches the appUcation manager determines 
which application is applicable to the user input, sends a request to a code generator for the 
application, and generates the application program code interface code required to access the 
information, which reads on "determining at least one method responsive to the command based 
on the current context." 

Additionally, at col. 4, lines 48-52, Johnson discloses a response generator that generates 
a response appropriate to the nature of the request submitted by the user and the current 
application, which reads on "executing the at least one method responsive to the command 
associated with the application." 
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Additionally, at col. 4, line 17-53 and col. 5, line 23-29, Johnson teaches "changing 
dialog focus," because the system ascertains that a different application should process the user 
request, and utilizes the different application's resources to process the request, since the 
procedures to access a new or different application requires the system to access or focus on the 
new or different application. The system of Johnson performs this processing without requiring 
the user to perform the task of accessing the application, but the application is indeed accessed 
and opened to allow for the processing of the user's request. 

Regarding claim 11, the combination of Johnson and Cote teach everything as claimed in 
claim 10. Additionally, at col. 3, lines 33-42, Johnson teaches the multi-modal interface is linked 
to the applications and allows for combining input from various modalities, which reads on 
"employing at least one multi-modal device for presenting the coromand." 

Regarding claim 12:, the combination of Johnson and Cote teaches everything as claimed 
in claim 11. Additionally, at col. 3, lines 37-42, Johnson discloses the input means can be via 
speech, typed or handwritten and includes point-and-click, touch and keyboard inputs, which 
reads on "multi-modal device is a computer." 

Regarding claim 13, the combination of Johnson and Cote teaches everything as claimed 
in claim 10. Johnson does not specifically teach providing a linked list of all events in the multi- 
modal history. However, providing a linked history of dialog between a user and a system was 
well known in the art. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or apphcation events (col. 7, line 9 to col. 9, line 53), which reads on 
"linked list of all events." 
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Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 

Regarding claim 14, the combination of Johnson and Cote teach everything as claimed in 
claim 13. Johnson does not specifically teach events in a multi-modal history include events 
linked by time, type, transaction, class or dialog focus. However, providing linked history events 
of a dialog between a user and a natural language system was well known. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or application events, and the events or records have chronological 
and/or hierarchical relationships (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 

Regarding claim 15, the combination of Johnson and Cote teach everything as claimed in 
claim 10. Additionally, at col. 5, line 55 continuing to col. 6, line 19, Johnson teaches 
referencing a concept/application table in which for a given input the application manager finds 
all concepts in the representation and determines each application registered with every concept 
in the representation to create an application set. The semantic representation is sent to the 
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applications code generator to allow the application to act on the code and retrieve the requested 
data, which reads on "referencing all active applications using a component control to determine 
the at least one method which is appropriate." 

Regarding claim 16, the combination of Johnson and Cote teach everything as claimed in 
claim 10. Additionally, at col. 6, lines 20-40, Johnson teaches the semantic representation of the 
users input is sent to the applications code generator to allow the application to act on the code 
and retrieve the requested data, which reads on "the conmiand is presented in a formal language 
such that a plurality of human utterances represent an action to be taken." 

Regarding claim 17, the combination of Johnson and Cote teaches everything as claimed 
in claim 11. Additionally, at col. 3, lies 60-62, Johnson teaches maintaining item in focus 
information, which reads on "maintaining a current dialog focus." At col. 5, line 61 continuing 
to col. 6, line 9, Johnson teaches concept/application table which reads on "a list of expected 
responses to provide a reference for determining the current context." 

Regarding claim 18, the combination of Johnson and Cote teach everything as claimed in 
claim 1 1 . Additionally, at col. 4, lines 9-17 and col. 5, lines 9-11, Johnson teaches the natural 
language processor interacts with the user to clarify missing information or ambiguities, which 
reads on "querying a user for one of information needed to resolve the current context and 
information needed to take an appropriate action. 

Regarding claim 19, Johnson discloses a multi-modal natural language interface for 
cross-application tasks which interprets user requests combining natural language input from the 
user with information selected from a current application and sends the request in the proper 
form to an appropriate auxihary application for processing. Additionally, at col. 3, lies 60-62, 
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Johnson teaches maintaining item in focus information, which reads on "maintaining a current 
dialog focus." At col. 5, line 61 continuing to col. 6, line 9, Johnson teaches concept/application 
table which reads on "a list of expected responses to provide a reference for determining the 
current context." 

Johnson fails to specifically disclose, "The appHcation associated with the command 
being one of a plurality of active applications." However, implementation of a computer system, 
which allows a user to interact with a plurality of applications using natural language, was well 
known in the art. 

In a similar field of endeavor, Cote discloses a system for controlling multiple user 
application programs by spoken input, which provides a speech recognition program working in 
conjunction with an operating system which provides multi-tasking capability and provides for 
implementation of a windowing system which facilitates the direction of user input to the various 
user application programs which are running (col. 3, lines 12-17; col. 4, lines 36-39). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to allow for the presented command to be associated with one 
of a plurahty of active appUcations, as suggested by Cote, for the purpose of allowing the user 
the convenience of accessing the processing of all applications to have a request processed 
automatically, without the user having to manually access any of the applications. 

Johnson fails to specifically disclose "the dialog manager determining current context of 
the command by reviewing a multi-modal history of events." However, determining current 
context of user input responses by reviewing a history of events was well known in the art. 



Application/Control Number: 09/374,374 Page 14 

Art Unit: 2654 

Cote teaches the system maintains a database for the appUcation programs that are 
running which allows the system to maintain an effective context for the user programs (col. 1, 
lines 47-49) and the system creates text events which are used to record information about all 
types of input events or appUcation events (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement determining current context of user input 
responses by reviewing a history of events as suggested by Cote, for the purpose of providing an 
improved system for controlling multiple user application programs by spoken input, as also 
suggested by Cote (col. 1, lines 32-34). 

Additionally, at col. 4, line 17-53 and col. 5, line 23-29, Johnson teaches "changing 
dialog focus," because the system ascertains that a different application should process the user 
request, and utilizes the different application's resources to process the request, since the 
procedures to access a new or different application requires the system to access or focus on the 
new or different application. The system of Johnson performs this processing without requiring 
the user to perform the task of accessing the application, but the application is indeed accessed 
and opened to allow for the processing of the user's request. 

Additionally, at col. 5, line 55 continuing to col. 6, line 19, Johnson teaches referencing a 
concept/application table in which for a given input the application manager finds all concepts in 
the representation and determines each application registered with every concept in the 
representation to create an application set. The semantic representation is sent to the applications 
code generator to allow the application to act on the code and retrieve the requested data, which 
reads on "a control component adapted to select at least one method responsive to the commands 
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received such that the system applies methods responsive to the commands for an appropriate 
application." 

Regarding claim 20, the combination of Johnson and Cote teach everything as claimed in 
claim 19. Additionally, at col. 4, lines 42-44, Johnson teaches the appropriate apphcation is an 
auxiliary application that is opened in the background. 

Regarding claim 21, the combination of Johnson and Cote teach everything as claimed in 
claim 19. Additionally, at col. 3, lines 37-42, Johnson discloses the input means can be via 
speech, typed or handwritten and includes point-and-click, touch and keyboard inputs, which 
reads on "multi-modal device is a computer." 

Regarding claim 22, the combination of Johnson and Cote teach everything as claimed in 
claim 19. Johnson does not specifically teach providing a linked list of all events in the multi- 
modal history. However, providing a linked history of dialog between a user and a system was 
well known in the art. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or application events (col. 7, line 9 to col. 9, line 53), which reads on 
"linked Ust of all events." 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 



Application/Control Number: 09/374,374 Page 16 

Art Unit: 2654 

Regarding claim 23, the combination of Johnson and Cote teach everything as claimed in 
claim 22. Johnson does not specifically teach events in a multi-modal history include events 
linked by time, type, transaction, class or dialog focus. However, providing linked history events 
of a dialog between a user and a natural language system was well known. 

Cote teaches the system creates text events, which are used to record information about 
all types of input events or application events, and the events or records have chronological 
and/or hierarchical relationships (col. 7, line 9 to col. 9, line 53). 

Therefore, it would have been obvious to one of ordinary skill at the time of the invention 
to modify the system of Johnson to implement a linked list of events in the multi-modal history, 
as suggested by Cote, for the purpose of removing the ambiguity of a users input query and for 
providing an improved system for controlling multiple user application programs by spoken 
input, as also suggested by Cote (col. 1, lines 32-34). 

Regarding claim 24, the combination of Johnson and Cote teach everything as claimed in 
claim 19. Additionally, at col. 5, line 55 continuing to col. 6, line 19, Johnson teaches 
referencing a concept/application table in which for a given input the application manager finds 
all concepts in the representation and determines each application registered with every concept 
in the representation to create an application set. The semantic representation is sent to the 
applications code generator to allow the application to act on the code and retrieve the requested 
data, which reads on "referencing all active applications using a component control to determine 
the at least one method which is appropriate." 

Regarding claim 25, the combination of Johnson and Cote teach everything as claimed in 
claim 24. Additionally, at col. 6, lines 20-40, Johnson teaches the semantic representation of the 
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users input is sent to the applications code generator to allow the appUcation to act on the code 



and retrieve the requested data, which reads on "the command is presented in a formal language 
such that a plurality of human utterances represent an action to be taken." 

Any inquiry conceming this communication or earlier communications from the 
examiner should be directed to Angela A Armstrong whose telephone number is 571-272-7598. 
The examiner can normally be reached on Monday-Thursday 1 1 :30-8:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on 571-272-7602. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Angela A Armstrong 

Examiner 

Art Unit 2654 



AAA 

May 16, 2005 




